Model Averaging With Holdout Estimation of the Posterior Distribution

The holdout estimation of the expected loss of a model is biased and noisy. Yet, practicians often rely on it to select the model to be used for further predictions. Repeating the learning phase with small variations of the training set, reveals a variation on the selected model which then induces an important variation of the final test performances. Thus, we propose a small modification to the k-fold crossvalidation that greatly enhances the generalization performances of the final predictor. Instead of using the empirical average of the validation losses to select a single model, we propose to use bootstrap to resample the validation losses (without retraining). The variations in the selected models induce a posterior distribution that is then used for model averaging. Comparing this novel approach to the classical cross-validation on 38 datasets with a significance test, shows that it has higher generalization performance with probability over 0.9.